V Annealing by Stochastic Neural Networks for Optimization

Two major classes of optimization techniques are the deterministic gradient methods and stochastic annealing methods. Gradient descent algorithms are greedy algorithms, which are subject to a fundamental limitation of being easily trapped in local minima of the cost function. Hopfield networks usually converge to a local minimum of energy function. Because of its deterministic input-output relation of the units in the network, the network is not able to escape from local minima. Although such a behavior may be desirable for an associative memory application, one usually needs to obtain the global optimum or a nearly optimum points for optimization applications. This problem is overcome by the use of stochastic annealing algorithms since they provide opportunity to escape from local minima. A highly attractive feature of the Boltzmann machine is its capability of escaping local minima through a relaxation technique based on simulated annealing [Hinton et al 83]. However, the use of simulated annealing is also responsible for an excessive computation time requirement that has hindered experimentation with the Boltzmann machine. Not only does simulated annealing require iterations at a sequence of temperatures that defines the annealing cycle but also each iteration requires many sweeps of its own. In order to overcome this major limitation of the Boltzmann machine a mean field approximation may be used, according to which the binary state stochastic neurons of the Boltzmann machine are replaced by deterministic mean values [Amit et al 85].